Display control device, display control system, and display control method

ABSTRACT

A captured image acquisition unit of a display control device according to the present invention acquires an image captured by an imaging device mounted on a loading vehicle. A loaded weight acquisition unit acquires a loaded weight measured by a weight scale mounted on a transport vehicle. A display image generation unit generates a display image obtained by disposing an image showing the loaded weight on the captured image. A display control unit outputs a display signal for displaying the display image to a display device.

TECHNICAL FIELD

The present invention relates to a display control device, a displaycontrol system, and a display control method of a display device in aremote operation device for performing remote operation of a loadingvehicle.

Priority is claimed on Japanese Patent Application No. 2018-163775,filed Aug. 31, 2018, the content of which is incorporated herein byreference.

BACKGROUND ART

A technique of performing remote operation of a loading vehicle isknown. In order to perform remote operation of the loading vehicle, itis necessary to be able to recognize a situation surrounding the loadingvehicle from the outside. For this reason, the loading vehicle performedremote operation includes an imaging device that images a surroundingsituation and a communication device that transmits the captured imageto the outside. Accordingly, an operator can perform operation whilevisually recognizing the image transmitted from the loading vehicle.

In addition, Patent Literature 1 discloses a technique of displaying theloadage of a transport vehicle on a display device of the loadingvehicle.

CITATION LIST Patent Literature Patent Literature 1

PCT International Publication No. WO2013/065415

SUMMARY OF INVENTION Technical Problem

In a case of performing remote operation of the loading vehicle, theoperator visually recognizes the transport vehicle from the imagedisplayed on a display. Thus, it is difficult to recognize the loadageof the transport vehicle.

An object of an aspect of the present invention is to provide a displaycontrol device, a display control system, and a display control method,which allow an operator of a loading vehicle related to remote controlto visually recognize the loadage of a transport vehicle easily.

Solution to Problem

A first aspect of the present invention provides a display controldevice for a display device. The display control device includes: acaptured image acquisition unit that is configured to acquire an imagecaptured by an imaging device mounted on a loading vehicle; a loadedweight acquisition unit that is configured to acquire a loaded weightmeasured by a weight scale mounted on a transport vehicle; a displayimage generation unit that is configured to generate a display imageobtained by disposing an image showing the loaded weight on the capturedimage; and a display control unit that is configured to output thedisplay image to the display device.

Advantageous Effects of Invention

In the at least one aspect, the display control device allows theoperator of the loading vehicle related to remote control to visuallyrecognize the loadage of the transport vehicle easily.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view illustrating a configuration of an operationsystem according to a first embodiment.

FIG. 2 is a diagram illustrating an example of a travel route.

FIG. 3 is an external view of a loading vehicle according to the firstembodiment.

FIG. 4 is an example of an image captured by an imaging device of theloading vehicle according to the first embodiment.

FIG. 5 is a schematic block diagram showing a configuration of acontrolling gear according to the first embodiment.

FIG. 6 is a flowchart showing an operation method of the controllinggear according to the first embodiment.

FIG. 7 is a schematic block diagram showing a configuration of a controldevice of a remote operation room according to the first embodiment.

FIG. 8 is a view illustrating an example of an image cut out from animage captured by a front camera.

FIG. 9 is a view illustrating an example of a display image displayed bya display device according to the first embodiment.

FIG. 10 is a flowchart showing a display control method implemented bythe control device for the remote operation room according to the firstembodiment.

DESCRIPTION OF EMBODIMENTS First Embodiment <<Remote Operation System>>

FIG. 1 is a schematic view illustrating a configuration of an operationsystem according to a first embodiment.

An operation system 1 includes one or a plurality of loading vehicles100 that operate through remote control, a plurality of transportvehicles 200, a controlling gear 300, an access point 400, and one or aplurality of remote operation rooms 500 that perform remote operation ofthe loading vehicles 100.

The transport vehicle 200 travels in an unmanned manner along a travelroute based on course data (for example, speed information andcoordinates for which an unmanned transport car is to head) receivedfrom the controlling gear 300. The transport vehicle 200 and thecontrolling gear 300 are connected to each other through communicationvia the access point 400. The controlling gear 300 acquires a positionand an azimuth direction from the transport vehicle 200 and generatescourse data used in the traveling of the transport vehicle 200 based onthe position and the azimuth direction. The controlling gear 300transmits the course data to the transport vehicle 200. The transportvehicle 200 travels in an unmanned manner based on the received coursedata. That is, the operation system 1 includes an unmanned transfersystem configured of the transport vehicle 200 and the controlling gear300. The operation system 1 according to the first embodiment includesthe unmanned transfer system, but some or all of transport cars may beoperated in a manned manner in another embodiment. In this case, it isnot necessary for the controlling gear 300 to transmit course data andan instruction related to loading, but the controlling gear acquires theposition and azimuth direction of the transport car.

The loading vehicle 100 is performed remote operation in response to anoperation signal transmitted from the remote operation room 500. Theloading vehicle 100 and the remote operation room 500 are connected toeach other through communication via the access point 400. A controldevice 550 of the remote operation room 500 receives operation of theloading vehicle 100 from an operator and transmits the operation signalto the loading vehicle 100. The loading vehicle 100 is driven inresponse to the operation signal received from the remote operation room500. That is, the operation system 1 includes a remote operating systemconfigured of the loading vehicle 100 and the remote operation room 500.

The loading vehicle 100 and the transport vehicle 200 operate at a worksite (for example, a mine and a quarry). The remote operation room 500is provided at a point separated away from the loading vehicle 100 andthe transport vehicle 200 (for example, a city and an inside of the worksite).

<<Travel Route>>

FIG. 2 is a diagram illustrating an example of a travel route.

A travel route R to be traveled is set for the transport vehicle 200.The travel route R has a connection route R1 which connects two areas A(for example, a loading site A1 and a dumping site A2) to each other andis determined in advance, an access route R2, an approach route R3, andan exit route R4, which are routes in each area A. The access route R2is a route that connects a standby point P1, which is one end of theconnection route R1 in each area A, and a predetermined turning point P2to each other. The approach route R3 is a route that connects theturning point P2 and a loading point P3 or a dumping point P4 to eachother in each area A. The exit route R4 is a route that connects theloading point P3 or the dumping point P4 and an exit point P5, which isthe other end of the connection route R1, to each other in each area A.The loading point P3 is a point set by operation by the operator of theloading vehicle 100. The turning point P2 is a point set by thecontrolling gear 300 according to the position of the loading point P3or the dumping point P4.

<<Remote Operation Room>>

The remote operation room 500 includes an operator's seat 510, a displaydevice 520, a first operation device 530, a second operation device 540,and the control device 550.

The display device 520 is disposed to face the operator's seat 510. Inthe embodiment, the term “face” refers to a state where the front of thedisplay device and the front of the operator's seat face each other soas to be parallel to each other or intersecting each other at an obtuseangle. The display device 520 is positioned in front of the operator'seyes when the operator sits in the operator's seat 510. As illustratedin FIG. 1, the display device 520 is configured of a central display521, a left display 522, a right display 523, an upper display 524, anda lower display 525, which are arranged side by side. The left display522 is provided on the left of the central display 521. The rightdisplay 523 is provided on the right of the central display 521. Theupper display 524 is provided above the central display 521. The lowerdisplay 525 is provided below the central display 521.

In another embodiment, the number of displays configuring the displaydevice 520 is not limited thereto. For example, the display device 520may be configured of one display. In addition, the display device 520may project an image on a curved surface or a spherical surface with aprojector.

The first operation device 530 is an operation device for the remoteoperating system. The first operation device 530 is positioned within arange where the operator can operate when the operator sits in theoperator's seat 510. The first operation device 530 includes, forexample, an electric lever and an electric pedal. The first operationdevice 530 receives inputs of a raising operation signal and loweringoperation signal of a boom 131, a pushing operation signal and pullingoperation signal of an arm 132, an excavating operation signal anddumping operation signal of a bucket 133, a swinging operation signal ofa swing body 120, and a traveling operation signal of a travel body 110.

The second operation device 540 is an operation device for the unmannedtransfer system. The second operation device 540 is positioned within arange where the operator can operate when the operator sits in theoperator's seat 510. The second operation device 540 is configured of atouch panel. The second operation device 540 receives inputs of aloading point instruction signal, an accessing instruction signal, and adeparture instruction signal for the transport vehicle 200. The secondoperation device 540 transmits the input operation signals to thecontrolling gear 300. The remote operation room 500 according to anotherembodiment may not include the second operation device 540.

The control device 550 causes the display device 520 to display an imageand vehicle body information which are received from loading vehicle100. That is, the control device 550 is an example of a display controldevice. In addition, the control device 550 transmits an operationsignal input in the first operation device 530 to the loading vehicle100.

<<Transport Vehicle>>

The transport vehicle 200 according to the first embodiment is anoff-road dump truck including a vessel. The transport vehicle 200according to another embodiment may be a transport car other than a dumptruck.

The transport vehicle 200 includes a position and azimuth directioncalculator 210, a payload meter 215, and a control device 220.

The position and azimuth direction calculator 210 calculates theposition and azimuth direction of the transport vehicle 200. Theposition and azimuth direction calculator 210 includes two receiversthat receive positioning signals from an artificial satellite thatconfigures Global Navigation Satellite System (GNSS). An example of GNSSis the Global Positioning System (GPS). The two receivers are providedat positions different from each other on the transport vehicle 200. Theposition and azimuth direction calculator 210 detects the position ofthe transport vehicle 200 in a site coordinate system based on thepositioning signals received by the receivers. The position and azimuthdirection calculator 210 uses the respective positioning signalsreceived by the two receivers to calculate an azimuth direction in whichthe transport vehicle 200 faces as a relationship between a provisionposition of one receiver and a provision position of the other receiver.Without being limited thereto, in another embodiment, for example, thetransport vehicle 200 may include an inertial measurement unit (IMU),and an azimuth direction may be calculated based on the measurementresult from the inertial measurement device. In this case, the drift ofthe inertial measurement unit may be corrected based on a travelingtrajectory of the transport vehicle 200. The transport vehicle 200according to another embodiment may not include the position and azimuthdirection calculator 210.

The payload meter 215 is a weight scale that measures the weight of aload loaded on the vessel of the transport vehicle 200, that is, theloaded weight of the transport vehicle 200. For example, the payloadmeter 215 may measure a pressure applied to a suspension cylinder of thetransport vehicle 200 and convert the pressure into the weight.

The control device 220 transmits the position and the azimuth direction,which are detected by the position and azimuth direction calculator 210,and the loaded weight measured by the payload meter 215 to thecontrolling gear 300. The control device 220 receives, from thecontrolling gear 300, course data, a dumping instruction, an accessinginstruction to the loading point, and a departure instruction from theloading point. The control device 220 causes the transport vehicle 200to travel according to the received course data, or moves the vessel ofthe transport vehicle 200 up and down according to the dumpinginstruction.

Operation signals to the transport vehicle 200 include an accessinginstruction signal and a departure instruction signal. The accessinginstruction signal is a signal for instructing the transport vehicle 200to access the loading point P3. The departure instruction signal is asignal for instructing the transport vehicle 200 to depart from theloading site A1 when the loading is completed.

In addition, the control device 220 transmits an arrival notification tothe controlling gear 300 when the transport vehicle 200 has arrived atthe loading point P3 in response to the accessing instruction signal.

<<Loading Vehicle>>

FIG. 3 is an external view of the loading vehicle according to the firstembodiment.

The loading vehicle 100 according to the first embodiment is a hydraulicexcavator. The loading vehicle 100 according to another embodiment maybe a loading vehicle other than the hydraulic excavator such as a wheelloader.

The loading vehicle 100 includes work equipment 130 that is driven by ahydraulic pressure, the swing body 120 that supports the work equipment130, and the travel body 110 that supports the swing body 120.

The work equipment 130 includes the boom 131, the arm 132, and thebucket 133. A base end portion of the boom 131 is attached to the swingbody 120 via a pin.

The arm 132 connects the boom 131 to the bucket 133. A base end portionof the arm 132 is attached to a tip portion of the boom 131 via a pin.

The bucket 133 includes a blade for excavating earth and a container foraccommodating the excavated earth. A base end portion of the bucket 133is attached to a tip portion of the arm 132 via a pin.

The swing body 120 includes a cab 121. The cab 121 is provided on theleft of the work equipment 130. The cab 121 is provided with a frontcamera 122. The front camera 122 is provided in an upper front portionin the cab 121. The front camera 122 captures an image of the front ofthe cab 121 through a windshield in a front portion of the cab 121.Herein, the “front” refers to a direction in which the work equipment130 is mounted on the swing body 120, and the “rear” refers to adirection opposite to the “front”. The “side” refers to a direction(right-and-left direction) intersecting the front-and-rear direction. Anexample of the front camera 122 includes an imaging device using acharge coupled device (CCD) sensor and a complementary metal oxidesemiconductor (CMOS) sensor. The loading vehicle 100 according toanother embodiment may not include the cab 121. Also in this case, thefront camera 122 is provided at a position corresponding to the cab 121to image the front. In addition, in another embodiment, the front camera122 may be configured of two or more cameras.

FIG. 4 is an example of an image captured by an imaging device of theloading vehicle according to the first embodiment. The front camera 122images a range where a loading operation target, which is in front ofthe work equipment 130 and the cab 121, appears. That is, in an image G1captured by the front camera 122, the loading operation target, which isin front of the work equipment 130 and the cab 121, appears asillustrated in FIG. 4. In addition, since the cab 121 is provided on theleft of the work equipment 130, part of the boom 131 appears in a rightportion of the image G1. In addition, a ceiling portion of the cab 121appears in an upper portion of the image G1.

The loading vehicle 100 includes the front camera 122, a bucket camera123, a position and azimuth direction calculator 124, an inclinationmeasuring instrument 125, and a control device 126.

The bucket camera 123 is provided on a front surface of the arm 132 andimages the inside of the bucket 133. Hereinafter, an image captured bythe bucket camera 123 will be referred to as a bucket internal image.The bucket camera 123 is an example of a sensor. The loading vehicle 100according to another embodiment may not include the bucket camera 123.

The position and azimuth direction calculator 124 calculates a positionof the swing body 120 and an azimuth direction in which the swing body120 faces. The position and azimuth direction calculator 124 includestwo receivers that receive positioning signals from an artificialsatellite that configures GNSS. The two receivers are provided atpositions different from each other on the swing body 120. The positionand azimuth direction calculator 124 detects a position of arepresentative point of the swing body 120 in a site coordinate system(the origin of an excavator coordinate system) based on the positioningsignals received by the receivers.

The position and azimuth direction calculator 124 uses the respectivepositioning signals received by the two receivers to calculate anazimuth direction in which the swing body 120 faces as a relationshipbetween a provision position of one receiver and a provision position ofthe other receiver.

In another embodiment, the position and azimuth direction calculator 124may detect an azimuth direction in which the swing body 120 faces basedon a measurement value of a rotary encoder or an IMU. In addition, theloading vehicle 100 according to another embodiment may not include theposition and azimuth direction calculator 124.

The inclination measuring instrument 125 measures the acceleration andangular speed of the swing body 120 and detects a posture (for example,a roll angle, a pitch angle, and a yaw angle) of the swing body 120based on the measurement result. The inclination measuring instrument125 is provided, for example, on a lower surface of the swing body 120.The inclination measuring instrument 125 can use, for example, aninertial measurement unit (IMU). In addition, the loading vehicle 100according to another embodiment may not include the inclinationmeasuring instrument 125.

The control device 126 transmits, to the remote operation room 500,information including an image captured by the front camera 122, animage captured by the bucket camera 123, and the swinging speed,position, azimuth direction, and inclination angle of the swing body120. Hereinafter, information which is measured by various sensorsincluded in the loading vehicle 100 and is transmitted by the controldevice 126 will be referred to as vehicle body information. The controldevice 126 receives an operation signal from the remote operation room500. The control device 126 drives the work equipment 130, the swingbody 120, or the travel body 110 based on the received operation signal.

<<Controlling Gear>>

FIG. 5 is a block diagram showing a configuration of the controllinggear according to the first embodiment.

The controlling gear 300 manages the traveling of the transport vehicle200.

The controlling gear 300 is a computer including a processor 3100, amain memory 3200, a storage 3300, and an interface 3400. The storage3300 stores a controlling program. The processor 3100 reads thecontrolling program from the storage 3300 to load the controllingprogram in the main memory 3200 and executes processing in accordancewith the controlling program. The controlling gear 300 is connected tocommunication means via the interface 3400.

The storage 3300 has storage areas as a travel route storage unit 3302and a vehicle information storage unit 3303. Examples of the storage3300 include a hard disk drive (HDD), a solid state drive (SSD), amagnetic disk, a magneto-optical disk, a compact disc read only memory(CD-ROM), a digital versatile disc read only memory (DVD-ROM), and asemiconductor memory. The storage 3300 may be an internal mediumdirectly connected to a common communication line of the controllinggear 300, or may be an external medium connected to the controlling gear300 via the interface 3400. The storage 3300 is a non-transitorytangible storage medium.

By correlating with the loading vehicle 100, the loading point storageunit 3301 stores identification information and coordinates of theloading point P3 at which the loading vehicle 100 performs loading.

The travel route storage unit 3302 stores the travel route R shown inFIG. 2 for each transport vehicle 200. The controlling gear 300 receivesa loading point instruction signal to prompt the operator of the loadingvehicle 100 to designate the coordinates of the loading point P3 andsets the loading point P3 in response to the loading point instructionsignal. The controlling gear 300 calculates the access route R2, theapproach route R3, and the exit route R4 each time the loading point P3is set. The loading point P3 may be designated by the operator of thetransport vehicle 200 who manages the transport vehicle 200 in a controlroom in which the controlling gear 300 is provided.

The vehicle information storage unit 3303 stores the positioninformation, azimuth direction information, loaded weight, maximumloaded weight, and standby information of each transport vehicle 200.The standby information is information indicating whether or not thetransport vehicle 200 is stopped at the loading point P3. That is, in acase where the transport vehicle 200 is stopped at the loading point P3,standby information indicates “true”, and in a case where the transportvehicle 200 is not at the loading point P3, standby informationindicates “false”.

By executing the controlling program, the processor 3100 includes avehicle information collection unit 3101, a traveling course generationunit 3102, an operation signal transmitting unit 3103, a notificationreceiving unit 3104, an update unit 3105, and a loaded weighttransmission unit 3106.

The vehicle information collection unit 3101 receives positioninformation, azimuth direction information, and loaded weightinformation from the transport vehicle 200 via the access point 400. Thevehicle information collection unit 3101 causes the vehicle informationstorage unit 3303 to store the received position information, azimuthdirection information, and loaded weight information.

The traveling course generation unit 3102, based on a travel route,which is stored in the travel route storage unit 3302, and positioninformation and azimuth direction information, which are stored in thevehicle information storage unit 3303, generates course data indicatingan area where the movement of the transport vehicle 200 is allowed andtransmits the course data to the transport vehicle 200. The course datais, for example, information indicating an area where the transportvehicle 200 can travel at a predetermined speed within a certain periodof time and does not overlap a travel route of another transport vehicle200. In a case where a stop signal of the transport vehicle 200 isreceived from the remote operation room 500, the traveling coursegeneration unit 3102 transmits the stop signal to the transport vehicle200.

The operation signal transmitting unit 3103 receives, from the remoteoperation room 500, an operation signal to the transport vehicle 200 andtransmits the operation signal to the transport vehicle 200. Theoperation signal to the transport vehicle 200 includes an accessinginstruction signal and a departure instruction signal. The accessinginstruction signal is a signal for instructing the transport vehicle 200to access the loading point P3. The departure instruction signal is asignal for instructing the transport vehicle 200 to depart from theloading site A1 when the loading is completed.

The notification receiving unit 3104 receives an arrival notificationfrom the transport vehicle 200.

In a case where an arrival notification from the transport vehicle 200is received, the update unit 3105 updates standby information correlatedwith the transport vehicle 200 in the vehicle information storage unit3303 to “true”. On the other hand, in a case where a departureinstruction signal is received from the remote operation room 500, theupdate unit 3105 updates standby information correlated with thetransport vehicle 200 in the vehicle information storage unit 3303 to“false”.

The loaded weight transmission unit 3106 transmits the loaded weight ofthe transport vehicle 200 to the remote operation room 500 for operatingthe loading vehicle 100 correlated with the loading point P3 of thetransport vehicle 200 whose standby information indicates “true”. Thatis, the loaded weight transmission unit 3106 transmits the loaded weightof a target transport vehicle, which is a loading target of the loadingvehicle 100, out of the plurality of transport vehicles 200, to theloading vehicle 100. The target transport vehicle of the loading vehicle100 is the transport vehicle 200 which is stopped at the loading pointP3 correlated with the loading vehicle 100 and of which standbyinformation indicates “true”.

<<Operation Method of Controlling Gear>>

FIG. 6 is a flowchart showing an operation method of the controllinggear according to the first embodiment.

The controlling gear 300 executes the following processing for eachpredetermined control cycle.

The vehicle information collection unit 3101 receives positioninformation, azimuth direction information, and loaded weightinformation from the transport vehicle 200 via the access point 400(Step S01). The vehicle information collection unit 3101 stores thereceived position information, azimuth direction information, and loadedweight information in the vehicle information storage unit 3303 (StepS02). In addition, the notification receiving unit 3104 determineswhether or not an arrival notification indicating that the transportvehicle has arrived at the loading point P3 in response to an accessinginstruction signal is received from the transport vehicle 200 (StepS03).

In a case where the arrival notification is received from the transportvehicle 200 (Step S03: YES), the update unit 3105 updates standbyinformation correlated with the transport vehicle 200 to “true” (StepS04). In a case where the arrival notification is not received from thetransport vehicle 200 (Step S03: NO), the standby information is notupdated.

Next, the traveling course generation unit 3102 generates course dataindicating an area where the movement of each transport vehicle 200 isallowed based on a travel route, which is stored in the travel routestorage unit 3302, and the position information and the azimuthdirection information, which are stored in the vehicle informationstorage unit 3303 (Step S05). The traveling course generation unit 3102transmits the generated course data to the transport vehicle 200 via theaccess point 400 (Step S06).

In addition, the operation signal transmitting unit 3103 receives, fromthe remote operation room 500, an operation signal to the transportvehicle 200 and transmits the operation signal to the transport vehicle200 (Step S07). Specifically, in a case where the accessing instructionsignal is received from the remote operation room 500, the operationsignal transmitting unit 3103 transmits the accessing instruction signalto the transport vehicle 200 stopped at the standby point P1, which is astarting point of the approach route R3 whose end point is the loadingpoint P3 correlated with the loading vehicle 100 operated by the remoteoperation room 500. In addition, in a case where a departure instructionsignal is received from the remote operation room 500, the operationsignal transmitting unit 3103 transmits the departure instruction signalto the transport vehicle 200 stopped at the loading point P3 correlatedwith the loading vehicle 100 operated by the remote operation room 500.The update unit 3105 determines whether or not the received operationsignal includes the departure instruction signal (Step S08). In a casewhere the operation signal includes the departure instruction signal(Step S08: YES), the update unit 3105 updates the standby informationcorrelated with the transport vehicle 200, which is a destination forthe departure instruction signal in the vehicle information storage unit3303, to “false” (Step S09).

In a case where the operation signal does not include the departureinstruction signal (Step S08: NO) or in a case where the standbyinformation is updated based on a transmission instruction signal inStep S09, the loaded weight transmission unit 3106 transmits the loadedweight and maximum loaded weight of the transport vehicle 200 to theremote operation room 500 for operating the loading vehicle 100correlated with the loading point P3 of the transport vehicle 200 whosestandby information indicates “true” (Step S10).

<<Control Device of Remote Operation Room>>

FIG. 7 is a schematic block diagram showing a configuration of thecontrol device for the remote operation room according to the firstembodiment.

The control device 550 is a computer including a processor 5100, a mainmemory 5200, a storage 5300, and an interface 5400. The storage 5300stores a program. The processor 5100 reads the program from the storage5300 to load the program in the main memory 5200, and executesprocessing in accordance with the program.

Examples of the storage 5300 include a hard disk drive (HDD), a solidstate drive (SSD), a magnetic disk, a magneto-optical disk, a compactdisc read only memory (CD-ROM), a digital versatile disc read onlymemory (DVD-ROM), and a semiconductor memory. The storage 5300 may be aninternal medium directly connected to a common communication line of thecontrol device 550, or may be an external medium connected to thecontrol device 550 via the interface 5400. The storage 5300 is anon-transitory tangible storage medium. In another embodiment, inaddition to the configuration or instead of the configuration, thecontrol device 550 may include a custom large scale integrated circuit(LSI) such as a programmable logic device (PLD) or a semi-LSI such asapplication specific integrated circuit (ASIC). Examples of the PLDinclude Programmable Array Logic (PAL), Generic Array Logic (GAL), acomplex programmable logic device (CPLD), and field programmable gatearray (FPGA). In this case, some or all of functions realized by theprocessor 5100 may be realized by the integrated circuit.

By executing the program, the processor 5100 includes a vehicle bodyinformation receiving unit 5101, an image cutout unit 5102, a loadedweight receiving unit 5103, a display image generation unit 5104, adisplay control unit 5105, and an operation signal transmission unit5106.

The vehicle body information receiving unit 5101 receives, from theloading vehicle 100, information including an image captured by thefront camera 122 and the swinging speed, position, azimuth direction,and inclination angle of the swing body 120. That is, the vehicle bodyinformation receiving unit 5101 is an example of a captured imageacquisition unit and a measurement value acquisition unit.

FIG. 8 is a view illustrating an example of an image cut out from animage captured by the front camera.

The image cutout unit 5102 cuts out each of a central image G11 fordisplaying on the central display 521, a left image G12 for displayingon the left display 522, a right image G13 for displaying on the rightdisplay 523, an upper image G14 for displaying on the upper display 524,and a lower image G15 for displaying on the lower display 525, from theimage G1 captured by the front camera 122, which is received by thevehicle body information receiving unit 5101. Each image is cut out suchthat the operator of the loading vehicle 100 easily recognizes thetopography of an excavation target and the position of the workequipment 130. In a case where the display device 520 is configured ofone display, the image cutout unit 5102 may not cut out an image.

The loaded weight receiving unit 5103 receives information including theloaded weight and maximum loaded weight of the transport vehicle 200from the controlling gear 300. The controlling gear 300 stores a maximumloaded weight for each transport vehicle 200 in the vehicle informationstorage unit 3303. Accordingly, the controlling gear 300 can recognizethe maximum loaded weight of each transport vehicle 200 even in a casewhere the plurality of transport vehicles 200 having different vehicleclasses travel. The loaded weight receiving unit 5103 is an example of aloaded weight acquisition unit.

FIG. 9 is a view illustrating an example of a display image displayed bythe display device according to the first embodiment.

The display image generation unit 5104 generates, for example, a rightimage for display G13 a by disposing a loaded weight image G2 showingthe loaded weight of the transport vehicle 200 received by the loadedweight receiving unit 5103 and a bucket internal image G3 received bythe vehicle body information receiving unit 5101 on the right image G13cut out by the image cutout unit 5102. The loaded weight image G2includes, for example, the loaded weight, maximum loaded weight, andremaining loaded weight of the transport vehicle 200. Although a sideview of the transport vehicle 200 is drawn in the loaded weight image G2of FIG. 9, the side view may be, for example, stored in advance in thestorage 5300.

The display control unit 5105 causes the central display 521 to displaythe central image G11. The control device 550 causes the left display522 to display the left image G12. The control device 550 causes theright display 523 to display the right image for display G13 a. Thecontrol device 550 causes the upper display 524 to display the upperimage G14. The control device 550 causes the lower display 525 todisplay the lower image G15.

The operation signal transmission unit 5106 generates an operationsignal based on the operation of the first operation device 530 by theoperator and transmits the operation signal to the loading vehicle 100.In addition, the operation signal transmission unit 5106 generates anoperation signal based on the operation of the second operation device540 by the operator and transmits the operation signal to thecontrolling gear 300.

<<Display Control Method of Remote Operation Room>>

FIG. 10 is a flowchart showing a display control method implemented bythe control device for the remote operation room according to the firstembodiment. When the loading vehicle 100 is started to be performedremote operation, the control device 550 executes the following displaycontrol in a predetermined cycle.

The vehicle body information receiving unit 5101 receives vehicle bodyinformation from the control device 126 of the loading vehicle 100 (StepS21). Next, the image cutout unit 5102 cuts out each of the centralimage G11, the left image G12, the right image G13, the upper image G14,and the lower image G15 from the image G1 captured by the front camera122 in the received vehicle body information (Step S22).

Next, the loaded weight receiving unit 5103 receives the loaded weightand maximum loaded weight of the transport vehicle 200 from thecontrolling gear 300 (Step S23). The display image generation unit 5104generates the right image for display G13 a by disposing the loadedweight image G2 showing the received loaded weight and maximum loadedweight and the bucket internal image G3 on the right image G13 (StepS24). The display control unit 5105 generates a display signal fordisplaying the central image G11, the left image G12, the right imagefor display G13 a, the upper image G14, and the lower image G15 on thedisplay device 520 and transmits the display signal to the displaydevice 520 (Step S25).

<<Workings and Effects>>

As described above, the control device 550 of the remote operation room500 according to the first embodiment acquires a loaded weight measuredin a target transport vehicle, which is a loading operation target ofthe loading vehicle 100, out of the plurality of transport vehicles 200,from the controlling gear 300 and causes the display device 520 todisplay the right image for display G13 a obtained by disposing theloaded weight image G2 on the right image G13. Accordingly, the controldevice 550 allows the operator of the loading vehicle 100 to easilyrecognize the loadage of the transport vehicle 200. In anotherembodiment, the loaded weight image G2 may be disposed on an image otherthan the right image G13.

In addition, the control device 550 of the remote operation room 500according to the first embodiment disposes the loaded weight image G2and the bucket internal image G3 captured by the bucket camera 123mounted on the loading vehicle 100 on the right image G13. Accordingly,the operator can recognize the loadage of the transport vehicle 200while checking the state of the loading vehicle 100. In particular, bymaking the bucket internal image G3 visually recognizable, the operatorin a remote control room can easily adjust the amount of load in thebucket 133. In another embodiment, the bucket internal image G3 may notbe included in the right image for display G13 a. The right image fordisplay G13 a may include an image showing a measurement value measuredby another sensor mounted on the loading vehicle 100 (for example, theinclination of the loading vehicle 100 and the posture of the workequipment 130).

In addition, in the operation system 1 according to the firstembodiment, the controlling gear 300 receives, from the transportvehicle 200, an arrival notification indicating the arrival to theloading point P3 and transmits the loaded weight related to thetransport vehicle 200 that is a transmitter of the arrival notification,to the remote operation room 500. Accordingly, the controlling gear 300can transmit the loaded weight measured in the target transport vehicle,which is a loading operation target of the loading vehicle 100, to theremote operation room 500. In another embodiment, the controlling gear300 may specify a target transport vehicle without depending on thearrival notification. For example, the controlling gear 300 according toanother embodiment may specify the transport vehicle 200 stopped at theloading point P3 near the loading vehicle 100 based on positioninformation of the transport vehicle 200. In addition, in anotherembodiment, the controlling gear 300 may transmit the positioninformation, azimuth direction information, and loaded weight of each ofall of the transport vehicles 200 to the remote operation room 500, andthe control device 550 of the remote operation room 500 may specify atarget transport vehicle. In addition, in another embodiment, thecontrol device 126 of the loading vehicle 100 may have some or all ofthe functions of the control device 550. In this case, the controldevice 126 may obtain payload information of the transport vehicle 200through inter-vehicle communication.

In addition, although the control device 550 of the remote operationroom 500 functions as the display control device for the display device520 in the first embodiment, the invention is not limited thereto. Forexample, in another embodiment, some functions of the control device 550may be realized by an external server device. In this case, the displaycontrol device is configured of the control device 550 and the externalserver device.

In addition, although the display control device is mounted on theoperation system 1 in the embodiment described above, the invention isnot limited thereto. For example, in another embodiment, the displaycontrol device may be applied to a radio control system that operatesthe loading vehicle 100 through wireless communication at a positionoutside the loading vehicle 100 where the loading vehicle 100 isvisually recognizable. In a case of being applied to the radio controlsystem, for example, a display device may be mounted on the controldevice.

INDUSTRIAL APPLICABILITY

The display control device according to the present invention allows theoperator of the loading vehicle related to remote control to easilyrecognize the loadage of the transport vehicle.

REFERENCE SIGNS LIST

-   1 Operation system-   100 Loading vehicle-   122 Front camera-   123 Bucket camera-   124 Position and azimuth direction calculator-   125 Inclination measuring instrument-   126 Control device-   200 Transport vehicle-   300 Controlling gear-   3101 Vehicle information collection unit-   3102 Traveling course generation unit-   3103 Operation signal transmitting unit-   3104 Notification receiving unit-   3105 Update unit-   3106 Loaded weight transmission unit-   3301 Loading point storage unit-   3302 Travel route storage unit-   3303 Vehicle information storage unit-   500 Remote operation room-   510 Operator's seat-   520 Display device-   530 First operation device-   540 Second operation device-   550 Control device-   5101 Vehicle body information receiving unit-   5102 Image cutout unit-   5103 Loaded weight receiving unit-   5104 Display image generation unit-   5105 Display control unit-   5106 Operation signal transmission unit

1. A display control device for a display device, the display controldevice comprising: a captured image acquisition unit that is configuredto acquire an image captured by an imaging device mounted on a loadingvehicle; a loaded weight acquisition unit that is configured to acquirea loaded weight measured by a weight scale mounted on a transportvehicle; a display image generation unit that is configured to generatea display image obtained by disposing an image showing the loaded weighton the captured image; and a display control unit that is configured tooutput the display image to the display device.
 2. The display controldevice according to claim 1, wherein the loaded weight acquisition unitacquires, from a controlling gear, the loaded weight measured by theweight scale mounted on a target transport vehicle, which is anoperation target of the loading vehicle, out of a plurality of thetransport vehicles that travel in an unmanned manner based on a travelinstruction transmitted from the controlling gear.
 3. The displaycontrol device according to claim 1, further comprising: a measurementvalue acquisition unit that is configured to acquire a measurement valuemeasured by a sensor mounted on the loading vehicle, wherein the displayimage generation unit generates a display image obtained by disposingthe image showing the loaded weight and an image showing the measurementvalue on the captured image.
 4. A display control system comprising: adisplay device; a captured image acquisition unit that is configured toacquire an image captured by an imaging device mounted on a loadingvehicle; a loaded weight acquisition unit that is configured to acquirea loaded weight measured by a weight scale mounted on a transportvehicle; a display image generation unit that is configured to generatea display image obtained by disposing an image showing the loaded weighton the captured image; and a display control unit that is configured tooutput the display image to the display device.
 5. A display controlmethod of a display device, the display control method comprising thesteps of: acquiring an image captured by an imaging device mounted on aloading vehicle; acquiring a loaded weight measured by a weight scalemounted on a transport vehicle; generating a display image obtained bydisposing an image showing the loaded weight on the captured image; anddisplaying the display image on the display device.